<html>
<head><meta charset="utf-8"><title>lets keep talking about perf.rlo · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html">lets keep talking about perf.rlo</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="225316827"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225316827" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225316827">(Feb 05 2021 at 15:58)</a>:</h4>
<p>New project group can spew stuff here</p>



<a name="225317004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317004">(Feb 05 2021 at 15:59)</a>:</h4>
<p>For future reference the meeting was at <a href="#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bdesign.20meeting.5D.202021-02-05.3A.20perf.2Erlo.20site/">https://rust-lang.zulipchat.com/#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bdesign.20meeting.5D.202021-02-05.3A.20perf.2Erlo.20site/</a></p>



<a name="225317087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317087">(Feb 05 2021 at 16:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> <span class="user-mention" data-user-id="352985">@tm</span> <span class="user-mention" data-user-id="224872">@rylev</span> <span class="user-mention" data-user-id="116122">@simulacrum</span> and <span class="user-mention" data-user-id="116083">@pnkfelix</span> all had their <span aria-label="raised hands" class="emoji emoji-1f64c" role="img" title="raised hands">:raised_hands:</span> raised</p>



<a name="225317127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317127">(Feb 05 2021 at 16:00)</a>:</h4>
<p>I seem to recall anp doing some investigation on the statistics front back when designing lolbench, à la automatic regression detection, I can't find what I'm thinking of rn but they may remember</p>



<a name="225317149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317149">(Feb 05 2021 at 16:00)</a>:</h4>
<p>What I'd like for us to do (among others' ideas):</p>
<ul>
<li>Gather pain points that impact the target audience (rustc devs who want to track the perf impact of their changes)</li>
<li>Understand the full flow of how these devs interact with perf.rlo and how the tool and perf team interact back (e.g., perf triage)</li>
<li>brainstorm ideas for how to improve this flow and address the painpoints</li>
</ul>



<a name="225317174"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317174" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317174">(Feb 05 2021 at 16:00)</a>:</h4>
<p><span class="user-mention silent" data-user-id="133247">bjorn3</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225317004">said</a>:</p>
<blockquote>
<p>For future reference the meeting was at <a href="#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bdesign.20meeting.5D.202021-02-05.3A.20perf.2Erlo.20site/">https://rust-lang.zulipchat.com/#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bdesign.20meeting.5D.202021-02-05.3A.20perf.2Erlo.20site/</a></p>
</blockquote>
<p>that's right, and the agenda/notes were largely this: <a href="https://hackmd.io/MqzuXVWnS22pVZgnFS5Tlg#collected-pain-points">https://hackmd.io/MqzuXVWnS22pVZgnFS5Tlg#collected-pain-points</a></p>



<a name="225317257"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317257" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317257">(Feb 05 2021 at 16:01)</a>:</h4>
<p>right, we can start with the current list of pain points and isolate the ones that impact target audience</p>



<a name="225317286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317286">(Feb 05 2021 at 16:01)</a>:</h4>
<p>separate them from the ones that don't (or have minimal impact to rustc devs)</p>



<a name="225317381"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317381" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317381">(Feb 05 2021 at 16:02)</a>:</h4>
<p><span class="user-mention" data-user-id="224872">@rylev</span> I'll make a fresh doc now, so that we don't clobber the notes from the meeting</p>



<a name="225317476"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317476" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317476">(Feb 05 2021 at 16:02)</a>:</h4>
<p>Yes, I don't want to <em>only</em> focus on the pain points listed though because it was a small group and we more than likely did not get a holistic accounting of pain points. But it's good to make sure those that were relevant and listed are addressed.</p>



<a name="225317488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317488">(Feb 05 2021 at 16:02)</a>:</h4>
<p>Categorized Pain Points: <a href="https://hackmd.io/5VHT_I8wRDebYVYoTeNK-Q">https://hackmd.io/5VHT_I8wRDebYVYoTeNK-Q</a></p>



<a name="225317536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317536">(Feb 05 2021 at 16:03)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225317476">said</a>:</p>
<blockquote>
<p>Yes, I don't want to <em>only</em> focus on the pain points listed though because it was a small group and we more than likely did not get a holistic accounting of pain points. But it's good to make sure those that were relevant and listed are addressed.</p>
</blockquote>
<p>Yep, we should definitely try to flesh out the list some more</p>



<a name="225317571"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317571" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317571">(Feb 05 2021 at 16:03)</a>:</h4>
<p>Beyond that getting a more holistic understanding of the current process and trying to decide how to improve that process would be a great thing to do</p>



<a name="225317611"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225317611" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225317611">(Feb 05 2021 at 16:03)</a>:</h4>
<p>we can try to take action in the meantime based on what we've identified</p>



<a name="225318069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225318069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Esteban Küber <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225318069">(Feb 05 2021 at 16:05)</a>:</h4>
<p>There are three things that would help with some of the identified issues:<br>
Allowing annotating spikes with text to manually point out the pr that caused it<br>
Auto running timer for all prs in a roll-up with regressions<br>
Giving a sense of scale for individual tests (medium swings in small tests are less critical) in the table overview</p>



<a name="225318320"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225318320" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225318320">(Feb 05 2021 at 16:07)</a>:</h4>
<p>Esteban, where were your hands when I asked for them!</p>



<a name="225318327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225318327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225318327">(Feb 05 2021 at 16:07)</a>:</h4>
<p>;)</p>



<a name="225318470"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225318470" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225318470">(Feb 05 2021 at 16:08)</a>:</h4>
<p>other possible pain points: </p>
<ul>
<li>unclear on how representative some of the current benchmarks are</li>
<li>coverage and scale: which parts of the compiler are "well covered" by the current benchmarks, and which are not; at which points do some pieces of rustc stop scaling</li>
<li>performance budget is limited, in that we are not easily able to add everything we'd like benchmarked</li>
<li>no data about IO, while incremental benchmarks hitting the disk more can be impactful, and hard to see</li>
<li>(maybe just a problem for me: possible lack of statistics / insight into the benchmarked crates, what and <em>why</em> they exercize what they do)</li>
</ul>



<a name="225318751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225318751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225318751">(Feb 05 2021 at 16:10)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> just as a follow up to the point about <code>@rust-timer queue</code>. We do still include in our target audience devs who <em>don't</em> run the timer but their PR is later found to be causing perf issues by the triage report, right?</p>



<a name="225319035"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225319035" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Esteban Küber <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225319035">(Feb 05 2021 at 16:12)</a>:</h4>
<blockquote>
<p>Esteban, where were your hands when I asked for them!</p>
</blockquote>
<p>I'm more of an ideas man 🤪</p>



<a name="225319077"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225319077" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Esteban Küber <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225319077">(Feb 05 2021 at 16:12)</a>:</h4>
<p>(want to help but not sure how much I can allocate)</p>



<a name="225319153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225319153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225319153">(Feb 05 2021 at 16:13)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225318751">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> just as a follow up to the point about <code>@rust-timer queue</code>. We do still include in our target audience devs who <em>don't</em> run the timer but their PR is later found to be causing perf issues by the triage report, right?</p>
</blockquote>
<p>Yes, that is true. I probably phrased it too narrowly. Being able to evaluate individual PR's is the point, not the mechanism by which one does that.</p>



<a name="225319833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225319833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225319833">(Feb 05 2021 at 16:17)</a>:</h4>
<p>/me is reaching the end of the day and has a stream on trait objects coming up but will continue this effort over the coming days</p>



<a name="225319938"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225319938" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225319938">(Feb 05 2021 at 16:18)</a>:</h4>
<p>I too have a packed day</p>



<a name="225348216"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225348216" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225348216">(Feb 05 2021 at 19:55)</a>:</h4>
<p>I've found that it's difficult to reproduce perf results locally - not just that it's noisy, but that I don't know the commands to run, and I'll end up opening PRs even though it takes longer just because it's easier</p>



<a name="225348364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225348364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225348364">(Feb 05 2021 at 19:56)</a>:</h4>
<p>And in general things other than <code>@rust-timer queue</code> don't seem to get much attention</p>



<a name="225359280"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225359280" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225359280">(Feb 05 2021 at 21:21)</a>:</h4>
<p>Pain point: not perf.rlo exactly, but...it's easy for regressions to sneak in, and it can take a long time before they get noticed or addressed (if ever!).</p>
<p>A small set of smoke-test benchmarks could be useful. The set could include a few benchmarks that are low-variance and good predictors of real regressions. Ideally, this would cover memory usage regressions too, so we'd need to devise a low variance but realistic benchmark for that.</p>
<p>Resources permitting, we could test all PRs against these before merging. At minimum, identification of a regression could be posted to the PR. Or we could block merging pending further investigation.</p>



<a name="225359530"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225359530" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225359530">(Feb 05 2021 at 21:24)</a>:</h4>
<blockquote>
<p>no data about IO, while incremental benchmarks hitting the disk more can be impactful, and hard to see</p>
</blockquote>
<p>It could make sense to record <code>/proc/pressure/*</code> output at the end of each benchmark. It distinguishes cpu, io and memory (paging) bottlenecks.</p>



<a name="225359644"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225359644" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225359644">(Feb 05 2021 at 21:24)</a>:</h4>
<p>/me googles <code>/proc/pressure/</code></p>



<a name="225359669"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225359669" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225359669">(Feb 05 2021 at 21:24)</a>:</h4>
<p><a href="https://lwn.net/Articles/759781/">https://lwn.net/Articles/759781/</a></p>



<a name="225360214"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225360214" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225360214">(Feb 05 2021 at 21:29)</a>:</h4>
<p>I use it to monitor things at my full-time. I wonder if it's going to be an insightful metric for batch jobs like what the compiler is.</p>



<a name="225360318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225360318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225360318">(Feb 05 2021 at 21:29)</a>:</h4>
<p>(in general having no pressure is better than having any, but the nature of batch jobs incur it almost by definition)</p>



<a name="225360673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225360673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225360673">(Feb 05 2021 at 21:32)</a>:</h4>
<p>One pain point that went unmentioned is us only measuring one t1 platform.</p>



<a name="225360703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225360703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225360703">(Feb 05 2021 at 21:33)</a>:</h4>
<p>CPU pressure could mean you're oversubscribing cores and could maybe size your thread pools better. significant IO pressure could hint at bad IO patterns. Generally I'm seeing low numbers when compiling rust locally, so things probably are fine already. But tracking it should be fairly cheap since you only need to snapshot it at the start and end of a job.</p>



<a name="225360795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225360795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225360795">(Feb 05 2021 at 21:33)</a>:</h4>
<p>Yeah that's very fair. Collecting it is super cheap (though requires a recent Linux kernel)</p>



<a name="225361028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225361028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225361028">(Feb 05 2021 at 21:35)</a>:</h4>
<p>Hm, is there a way to get it for a cgroup/process group and not system-wide?</p>



<a name="225361104"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225361104" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225361104">(Feb 05 2021 at 21:36)</a>:</h4>
<p>yes, it's somewhere in the cgroup virtual filesystem</p>



<a name="225361186"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225361186" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225361186">(Feb 05 2021 at 21:37)</a>:</h4>
<blockquote>
<p>In systems where control groups are in use, there will also be a set of files (cpu.pressure, memory.pressure, and io.pressure) associated with each group.</p>
</blockquote>



<a name="225593698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225593698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225593698">(Feb 08 2021 at 19:18)</a>:</h4>
<p><span class="user-mention" data-user-id="224872">@rylev</span> should we try to schedule some sort of regular synchronization point? I don't know what cadence would make the most sense; I just want to keep up the same kind of energy that we observed on Friday.</p>



<a name="225593896"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225593896" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225593896">(Feb 08 2021 at 19:19)</a>:</h4>
<p>Yes, I’m relatively free in the morning and early afternoon Europe time.</p>



<a name="225593978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225593978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225593978">(Feb 08 2021 at 19:20)</a>:</h4>
<p>I made a tiny bit of progress on the doc today but was going to dive in fully tomorrow</p>



<a name="225597033"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225597033" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225597033">(Feb 08 2021 at 19:44)</a>:</h4>
<p>Simple action item: Change page layout so that the metric selector is consistently on the top of the page. <span aria-label="upside down" class="emoji emoji-1f643" role="img" title="upside down">:upside_down:</span></p>



<a name="225599291"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225599291" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225599291">(Feb 08 2021 at 20:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> you and I briefly discussed "where is the perf data stored" last week, but I didn't dive in and get sufficient info to e.g. grab the data myself. I do see <a href="https://github.com/rust-lang/rustc-perf/pull/840/files">rustc-perf PR #840</a>, which says we no longer publish the database. Is that because its too expensive to serve it up to the world? Or some other reason?</p>



<a name="225599398"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225599398" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225599398">(Feb 08 2021 at 20:01)</a>:</h4>
<p>Not too expensive to <em>serve</em> but rather impractical to export. The current tooling took more than 30 minutes (albeit in a small ec2 machine), and that was a good 1/2-3/4 year ago I think</p>



<a name="225599541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225599541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225599541">(Feb 08 2021 at 20:02)</a>:</h4>
<p>for things like changing page layout or really anything about the frontend, you can generally just point the API queries at <a href="http://perf.rust-lang.org">perf.rust-lang.org</a> and that should work fine</p>



<a name="225599629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225599629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225599629">(Feb 08 2021 at 20:03)</a>:</h4>
<p>Yeah my Q about DB storage was unrelated to the layout thing; it was more the result of me skimming over the website</p>



<a name="225599734"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225599734" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225599734">(Feb 08 2021 at 20:03)</a>:</h4>
<p>A more serious suggestion I have in mind: Reduce the speed-bumps between a benchmark result (i.e. a specific datum for a given benchmark+metric) and how to perform the same experiment locally (maybe to point of providing a command line invocation that should spit out the metric for one's local host).</p>



<a name="225601252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225601252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225601252">(Feb 08 2021 at 20:17)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225599541">said</a>:</p>
<blockquote>
<p>you can generally just point the API queries at <a href="http://perf.rust-lang.org">perf.rust-lang.org</a> and that should work fine</p>
</blockquote>
<p>does that require having the results database ?</p>



<a name="225601396"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225601396" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225601396">(Feb 08 2021 at 20:18)</a>:</h4>
<p>no, you edit the js here - <a href="https://github.com/rust-lang/rustc-perf/blob/master/site/static/shared.js#L1">https://github.com/rust-lang/rustc-perf/blob/master/site/static/shared.js#L1</a> - to <a href="http://perf.rust-lang.org/perf">perf.rust-lang.org/perf</a></p>



<a name="225601547"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225601547" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225601547">(Feb 08 2021 at 20:19)</a>:</h4>
<p>and now I remember I've done this before to test the compare links ...</p>



<a name="225602907"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225602907" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225602907">(Feb 08 2021 at 20:28)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225597033">said</a>:</p>
<blockquote>
<p>Simple action item: Change page layout so that the metric selector is consistently on the top of the page. <span aria-label="upside down" class="emoji emoji-1f643" role="img" title="upside down">:upside_down:</span></p>
</blockquote>
<p>you got it <a href="https://github.com/rust-lang/rustc-perf/pull/841">https://github.com/rust-lang/rustc-perf/pull/841</a></p>



<a name="225609463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225609463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225609463">(Feb 08 2021 at 21:16)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> <span class="user-mention" data-user-id="224872">@rylev</span> I think what would be most useful for me is to get some sense of what would be helpful from me to get this effort off the ground; let me know if e.g. a sync call would be good or similar. I can also write up some architecture/design docs on perf.rlo today but it would take some time and it's not <em>that</em> big so maybe not too useful.</p>



<a name="225610326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225610326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225610326">(Feb 08 2021 at 21:23)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225599734">said</a>:</p>
<blockquote>
<p>A more serious suggestion I have in mind: Reduce the speed-bumps between a benchmark result (i.e. a specific datum for a given benchmark+metric) and how to perform the same experiment locally (maybe to point of providing a command line invocation that should spit out the metric for one's local host).</p>
</blockquote>
<p>I've had an idea that's somewhat along these lines (but might also tip-toe over the line into off-topic):</p>
<p>You could pick some ten year old desktop spec, emulate that in all the ways that count, and then run your benchmarks in that - not everyone is going to have fast local hardware, so benchmarking natively produces results that aren't comparable, but as long as your hardware is fast enough to emulate our desired slow spec, you can produce comparable benchmark results locally. Depending how you package that up, I could imagine the portability resembling that of a docker container, because there will necessarily be some surrounding environment that you'd normally want to avoid benchmarking natively (in contrast, setting up perf locally requires system dependencies that sys crates require, etc, there's more work). I'm not necessarily seriously proposing this, because it would likely be to slower to produce benchmark results, which is undesirable, and it could be unrealistic (we might want to check that we can utilize today's hardware fully), but I think it's a fun idea.</p>



<a name="225610617"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225610617" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225610617">(Feb 08 2021 at 21:25)</a>:</h4>
<p>I think this is basically what cachegrind/valgrind does, though that's too much a slowdown to really be viable I suspect. :)</p>



<a name="225610807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225610807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225610807">(Feb 08 2021 at 21:26)</a>:</h4>
<p>Like Iai? <a href="https://bheisler.github.io/post/criterion-rs-0-3-4/">https://bheisler.github.io/post/criterion-rs-0-3-4/</a></p>



<a name="225610972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225610972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225610972">(Feb 08 2021 at 21:28)</a>:</h4>
<p>Huh, interesting - I had no idea. I'll need to read into that.</p>



<a name="225611621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225611621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225611621">(Feb 08 2021 at 21:33)</a>:</h4>
<p>Yeah, iai is based on valgrind under the hood I think. But indeed.</p>



<a name="225626094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225626094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225626094">(Feb 08 2021 at 23:42)</a>:</h4>
<p>Since instruction counts are more stable than wall time is there a reason why the bootstrap timings are not gathered via perf stat? At least the totals.</p>



<a name="225626271"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225626271" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225626271">(Feb 08 2021 at 23:45)</a>:</h4>
<p>Not really; I wanted to get something up quickly and the existing statistical instrumentation did not map cleanly to the needs of bootstrap collection - in particular, the timing info is currently extracted from built-in support in rustc (print-step-timings) rather than perf stat or similar.</p>



<a name="225701996"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225701996" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225701996">(Feb 09 2021 at 15:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> I took some time to analyze the feedback, document the current performance process and ask questions about how we can improve it: <a href="https://hackmd.io/5VHT_I8wRDebYVYoTeNK-Q#Feedback-Analysis">https://hackmd.io/5VHT_I8wRDebYVYoTeNK-Q#Feedback-Analysis</a> We should discuss this to make sure we're capturing everything, and then we can brainstorm ideas of how to improve the process.</p>



<a name="225702337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225702337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225702337">(Feb 09 2021 at 15:02)</a>:</h4>
<p>I have many ideas on how to improve things, but I'd like to hold off on sharing them until we've got a collective understanding of the current process and its weak points</p>



<a name="225704604"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225704604" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225704604">(Feb 09 2021 at 15:17)</a>:</h4>
<blockquote>
<p>This question relates to a lot of the feedback about it being hard to interpret the metrics and to know when metrics represent a “true” regression.</p>
</blockquote>
<p>Knowing the variance could help. Maybe the noise-run could be augmented by running the short-running ones in a loop to get enough samples. Or manually trigger multiple iterations for ones that are suspected to be noisy. Something like that.</p>



<a name="225708624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225708624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225708624">(Feb 09 2021 at 15:42)</a>:</h4>
<p>The instruction counts a quite precise, I think, but a large bias is built into the compiler, due to unpredictable effects of CGU partitioning. One recent measurement: a single extra call in rustc_privacy crate changed partitioning of 10% of mono-items in the crate. This indirect impact is often larger than direct one of code changes.</p>



<a name="225711905"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225711905" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225711905">(Feb 09 2021 at 16:02)</a>:</h4>
<p>I've wondered a <em>lot</em> about our CGU partitioning algorithm; right now its pretty heavily tilted towards one particular goal (namely estimated load balancing). I suspect there are other partitions that would not have quite such a heavy level of unpredictability w.r.t. the performance impact of code changes.</p>



<a name="225713576"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225713576" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225713576">(Feb 09 2021 at 16:13)</a>:</h4>
<p>to bring this back to the topic at hand: Other metrics we probably should be gathering is about the CGU partitioning: How many parts, what were their estimated sizes, and what were their "actual" sizes (whatever "actual" may mean; bitcode size is one obvious choice; actual object code size is another...)</p>



<a name="225714285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225714285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225714285">(Feb 09 2021 at 16:17)</a>:</h4>
<p>I'd like to break this problem into many parts that loosely follow the chronology of performance tracking as it currently stands: </p>
<ul>
<li>How do we improve the metrics we're actually gathering? </li>
<li>How do we make sure the information is actually generated and looked at? (i.e., it's possible to just never worry about performance)</li>
<li>How do we ensure that that information is interpreted correctly?</li>
<li>How do we ensure that information is acted upon?</li>
<li>How do we ensure regressions are fixed (or at least explicitly decided as "won't fix")</li>
</ul>



<a name="225714615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225714615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225714615">(Feb 09 2021 at 16:18)</a>:</h4>
<p>The conversation is currently geared heavily towards the first item, but I don't want us to forget about the others.</p>



<a name="225714734"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225714734" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225714734">(Feb 09 2021 at 16:19)</a>:</h4>
<p>i've been wondering if an entirely different end, e.g. one based on Jupyter, would allow us to put more power into the hands of people viewing the site.</p>



<a name="225714916"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225714916" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225714916">(Feb 09 2021 at 16:20)</a>:</h4>
<p>(I guess that <em>might</em> fall under the bullet of "how do we ensure the information is interpreted correctly". Though I suspect it may belong in a distinct bullet: How do we give the audience more power in how they can manipulate the dataset.)</p>



<a name="225715392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225715392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225715392">(Feb 09 2021 at 16:23)</a>:</h4>
<p>Perhaps I'm missing a "discover the underlying cause" which is different than answering the question "is there actually a regression happening". I'd prefer to try to keep things more focused on the problem we're trying to solve rather than focusing too much on the tools themselves.</p>



<a name="225715547"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225715547" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225715547">(Feb 09 2021 at 16:24)</a>:</h4>
<p>It's been my personal experience that once it's been identified that a regression is happening, finding out the cause is not often the hard part (though sometimes it is, so I'm not discounting the idea).</p>



<a name="225715956"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225715956" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225715956">(Feb 09 2021 at 16:27)</a>:</h4>
<p>I'll say it another way, I'm personally avoiding offering solutions right now, because I really want us to get a better collective understanding of the problem set.</p>



<a name="225719536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225719536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225719536">(Feb 09 2021 at 16:50)</a>:</h4>
<p><span class="user-mention silent" data-user-id="352985">tm</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225708624">said</a>:</p>
<blockquote>
<p>The instruction counts a quite precise, I think</p>
</blockquote>
<p>"precise, I think" doesn't help during reviews. Actual numbers how large the variance on individual benchmarks is would be nice. Is 0.1% significant on that particular one or not?</p>
<blockquote>
<p>but a large bias is built into the compiler, due to unpredictable effects of CGU partitioning.</p>
</blockquote>
<p>opt build tests run with CGUs &gt; 1 too?</p>



<a name="225723686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225723686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> tm <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225723686">(Feb 09 2021 at 17:14)</a>:</h4>
<p>I was referring to rustc specifically, which would use the default 16 CGUs for the compiler crates, one CGU for the standard library crates, and ∞ for compiler builtins. There is of course similar aspect with respect to benchmarks, but you would have to consult their Cargo.toml to see what they do exactly (usually defaults of 16 / 256 incremental).</p>



<a name="225725579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225725579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225725579">(Feb 09 2021 at 17:27)</a>:</h4>
<p><span class="user-mention" data-user-id="330154">@The 8472</span> <span class="user-mention" data-user-id="352985">@tm</span> Would you be able to move this particular conversation to a new topic? That will help make sure this topic stays focused on the more "meta" point.</p>



<a name="225727658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225727658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Eh2406 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225727658">(Feb 09 2021 at 17:41)</a>:</h4>
<p>Big +1 for not looking at solutions until the problem space has been explored. But I was involved in developing some rudimentary statistical models of regressions in Lolbench, and would love to be involved in similar efforts for perf. So keep me in mind if that solution would be valuable.</p>



<a name="225812715"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225812715" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225812715">(Feb 10 2021 at 09:40)</a>:</h4>
<p>I've got some feedback on a few points here and in <a href="https://hackmd.io/5VHT_I8wRDebYVYoTeNK-Q#Feedback-Analysis">https://hackmd.io/5VHT_I8wRDebYVYoTeNK-Q#Feedback-Analysis</a>:</p>
<blockquote>
<ul>
<li>the compare mode presentation, for each benchmark, has one "full" and then several incr variants. This might be leading to over-emphasis on incr performance.</li>
</ul>
</blockquote>
<p>There are three main workloads caused by incr. comp. which are rather different from each other, and are important to keep track of:</p>
<ul>
<li>starting from an empty cache -&gt; important if we want incr. comp. to be the default (as it is atm for debug and check builds)</li>
<li>re-building after a small(ish) change -&gt; probably the most common case</li>
<li>re-compiling with no changes present -&gt; this happens if an upstream dependency changes</li>
</ul>



<a name="225813075"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813075" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813075">(Feb 10 2021 at 09:43)</a>:</h4>
<p>I think it's a good idea to keep track of (at least some variation) of each of these three. However, for some crates with have a lot of different cases (because 'small change' can mean a myriad of different things). We could trim those down</p>



<a name="225813147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813147">(Feb 10 2021 at 09:44)</a>:</h4>
<p><em>How</em> we present the different incr. cases is also a different question</p>



<a name="225813185"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813185" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813185">(Feb 10 2021 at 09:44)</a>:</h4>
<p>we could do something visually to make them less prominent</p>



<a name="225813283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813283">(Feb 10 2021 at 09:45)</a>:</h4>
<p>and it would definitely be a good idea to provide descriptions for each benchmark, i.e. why it's there and what it measures</p>



<a name="225813451"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813451" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813451">(Feb 10 2021 at 09:47)</a>:</h4>
<blockquote>
<ul>
<li>Not clear which benchmarks matter. Am I supposed to weight all equally? (Does summary weight all equally?)</li>
</ul>
</blockquote>
<p>Yes, we have lots of "stress tests" for particular code paths. I'm not quite sure what to make of those. They can definitely be useful to spot some regressions, but I'm not sure if they should be included in the weighted average</p>



<a name="225813487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813487">(Feb 10 2021 at 09:47)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> Shall we schedule a meeting to talk over these points? I'm fairly open today earlier in the day (i.e., for the next 6 hours)</p>



<a name="225813589"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225813589" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225813589">(Feb 10 2021 at 09:48)</a>:</h4>
<p>historically we tried to capture "high impact" crates, i.e. crates that are used by many projects. I think it's a good idea to have commonly used crates in the standard suite.</p>



<a name="225814712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225814712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225814712">(Feb 10 2021 at 10:00)</a>:</h4>
<blockquote>
<ul>
<li>noise, in particular on some benchmarks, is frequently present, and makes it easy to essentially ignore such benchmarks.</li>
</ul>
</blockquote>
<p>Yup, noise often makes it hard to interpret benchmarks, and the smaller a benchmark the bigger the noise, usually. One problematic workaround to the noise problem is that people usually just look at the less noisy metrics (i.e. mostly instruction count). But we actually only care about wall-time and memory (EDIT: and disk) usage in practice (incidentally the two noisiest metrics <span aria-label="upside down" class="emoji emoji-1f643" role="img" title="upside down">:upside_down:</span>) I have definitely seen cases where instruction count did not change, but wall time did (and significantly so) because of CPU caching effects, synchronization, or blocking I/O.</p>
<p>The most reliable way of reducing noise (that I know of) is doing more benchmark iterations. Given that benchmarking parallelizes well, this is mostly a matter of resources. Reducing noise would go a long way in helping folks correctly interpret results.</p>
<p>Another thing to look at would be the configuration of the benchmarking hardware. We probably don't want to measure how well the cooling system enables turbo boost and such <span aria-label="smile" class="emoji emoji-1f642" role="img" title="smile">:smile:</span> </p>
<p>Also, are there ways of making memory consumption less noisy, e.g. by using a different (but realistic) allocator?</p>



<a name="225814830"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225814830" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225814830">(Feb 10 2021 at 10:01)</a>:</h4>
<blockquote>
<p>A small set of smoke-test benchmarks could be useful. The set could include a few benchmarks that are low-variance and good predictors of real regressions. </p>
</blockquote>
<p>This is a great idea!</p>



<a name="225815305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225815305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225815305">(Feb 10 2021 at 10:06)</a>:</h4>
<p>Despite what I said about instruction counts above, they can be very useful since they don't depend on the underlying hardware much. So, for a smoke-test it might actually be feasible to run that in a VM, if it only measures instruction counts. Memory consumption should also be mostly hardware independent, so that could be measured too. Given a low enough number of test cases and a high iteration count, it sounds like it should be possible to catch prominent regressions in a reasonable amount of time.</p>



<a name="225816468"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225816468" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225816468">(Feb 10 2021 at 10:19)</a>:</h4>
<blockquote>
<p>The conversation is currently geared heavily towards the first item, but I don't want us to forget about the others.</p>
</blockquote>
<p>I want to second that statement. It's easy to get lost in technical details (when it's often simple UI improvements that make a big difference).</p>



<a name="225823851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225823851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225823851">(Feb 10 2021 at 11:38)</a>:</h4>
<p><span class="user-mention silent" data-user-id="124287">mw</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225814712">said</a>:</p>
<blockquote>
<p>Also, are there ways of making memory consumption less noisy, e.g. by using a different (but realistic) allocator?</p>
</blockquote>
<p>The noise here comes from the fact that the compilation is multi-threaded and thus allocations to a global heap are done interleaved, etc. I imagine that memory consumption could be measured much more precisely in a single-threaded environment; or if the allocator implementation maintained per-thread heaps (though timing would still introduce non-determinism).</p>
<p>All that said I don't think we can replace (easily?) the allocator llvm uses.</p>



<a name="225848348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225848348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225848348">(Feb 10 2021 at 15:01)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225813487">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="116083">pnkfelix</span> Shall we schedule a meeting to talk over these points? I'm fairly open today earlier in the day (i.e., for the next 6 hours)</p>
</blockquote>
<p>I didn’t see this until now</p>



<a name="225848404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225848404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225848404">(Feb 10 2021 at 15:01)</a>:</h4>
<p><span class="user-mention" data-user-id="224872">@rylev</span> I’ve got a meeting in 29 minutes. Do you want to try to chat quickly now?</p>



<a name="225848790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225848790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225848790">(Feb 10 2021 at 15:04)</a>:</h4>
<p>Otherwise, I’ll be busy until 12pm EST. (the <code>&lt;time</code> macro is not working for me at the moment in the app…)</p>



<a name="225868785"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225868785" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225868785">(Feb 10 2021 at 16:45)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> sorry just saw this. I'm free in 15 minutes (noon EST)</p>



<a name="225871894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225871894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225871894">(Feb 10 2021 at 17:03)</a>:</h4>
<p>hi hi</p>



<a name="225871926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225871926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225871926">(Feb 10 2021 at 17:03)</a>:</h4>
<p>I'll just make a zoom mtg and post the invite here</p>



<a name="225872105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225872105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225872105">(Feb 10 2021 at 17:04)</a>:</h4>
<p><a href="https://zoom.us/j/99832269996">https://zoom.us/j/99832269996</a></p>



<a name="225872122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225872122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225872122">(Feb 10 2021 at 17:04)</a>:</h4>
<p>Passcode: 384375</p>



<a name="225872334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225872334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225872334">(Feb 10 2021 at 17:06)</a>:</h4>
<p><span class="user-mention" data-user-id="224872">@rylev</span> ^</p>



<a name="225966305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/225966305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#225966305">(Feb 11 2021 at 09:31)</a>:</h4>
<p><span class="user-mention" data-user-id="123586">@nagisa</span> Yeah, those are both good points.</p>



<a name="226049079"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226049079" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226049079">(Feb 11 2021 at 19:59)</a>:</h4>
<p><span class="user-mention silent" data-user-id="123586">nagisa</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/lets.20keep.20talking.20about.20perf.2Erlo/near/225823851">said</a>:</p>
<blockquote>
<p>All that said I don't think we can replace (easily?) the allocator llvm uses.</p>
</blockquote>
<p>I hadn’t thought carefully about that detail. But I also wonder: Maybe that’s okay? I.e., I imagine we could still get very useful feedback even if we solely gather data based on swapping the Rust allocator alone...</p>



<a name="226052522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226052522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226052522">(Feb 11 2021 at 20:25)</a>:</h4>
<p>Isn't peak rss in majority of cases somewhere during LLVM codegen?</p>



<a name="226052706"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226052706" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226052706">(Feb 11 2021 at 20:27)</a>:</h4>
<p>Yes (I'm working in this area on this very issue)</p>



<a name="226053254"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226053254" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226053254">(Feb 11 2021 at 20:31)</a>:</h4>
<p>(that said, maybe it would make sense to collect peak-rss at various points of compilation and plot it somehow?)</p>



<a name="226054141"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226054141" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226054141">(Feb 11 2021 at 20:38)</a>:</h4>
<p>Funny you should mention that :)</p>
<p><a href="/user_uploads/4715/-fH4ZMOD2SzettAUNx1EPnPE/codegen_comparison.png">codegen_comparison.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/-fH4ZMOD2SzettAUNx1EPnPE/codegen_comparison.png" title="codegen_comparison.png"><img src="/user_uploads/4715/-fH4ZMOD2SzettAUNx1EPnPE/codegen_comparison.png"></a></div><p>This was hacked together by reading RSS basically every loop iteration of the codegen coordinator loop.</p>



<a name="226054450"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226054450" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Tyson Nottingham <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226054450">(Feb 11 2021 at 20:41)</a>:</h4>
<p>But yeah, doing it for real with say, <code>-Z self-profile</code> would be nice</p>



<a name="226314139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226314139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226314139">(Feb 14 2021 at 16:55)</a>:</h4>
<p>Building rustc when measuring perf takes a lot of time. Would it be possible (and accurate enough) to measure the bootstrap time of the try build instead of rebuilding rustc once again?</p>



<a name="226314194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226314194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226314194">(Feb 14 2021 at 16:56)</a>:</h4>
<p>doesn't perf.rlo already record the bootstrap time?</p>



<a name="226315283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226315283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226315283">(Feb 14 2021 at 17:19)</a>:</h4>
<p>My impression is that it records it by rebuilding rustc once again. This is what I would like to avoid.</p>



<a name="226318797"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226318797" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226318797">(Feb 14 2021 at 18:45)</a>:</h4>
<p>No, try builds are highly variable. Bootstrap timing on perf is accurate to sub-percent in total and just a few percent on individual crates.</p>



<a name="226554703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226554703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226554703">(Feb 16 2021 at 18:24)</a>:</h4>
<p>Possible enhancement : allow to select the metric in the detailed query panel. The compare panel often focuses on instruction count, while the detailed panel computes wall-time. It makes difficult to track where in the compiler the regression happens.</p>



<a name="226672081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/lets%20keep%20talking%20about%20perf.rlo/near/226672081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/lets.20keep.20talking.20about.20perf.2Erlo.html#226672081">(Feb 17 2021 at 15:24)</a>:</h4>
<p>FYI: I'm working on a document which aims to help us improve the performance tracking/triage process. The document will be purely aimed at the <em>process</em> so things like how to improve perf.rlo itself won't be discussed. I hope to have this done sometime today or tomorrow</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>